Easy Dell PowerEdge Firmware Updates, 2019 Edition

Bob Plankers March 8, 2019

Best Practices, Data Center, Hardware, How To, Security, System Administration, Virtualization, VMware

I’ve become quite the minimalist in my environments, mostly because I’ve been doing a lot of compliance & security work. Speaking generally, most hardware management tools don’t & won’t pass any form of compliance audit and in that context are way more trouble than they’re worth (negative ROI, see my post “Free, Like a Puppy“). I tend to skip anything that isn’t delivered directly as part of the OS from the OS vendor. It’s a complication I don’t need or want. I use this same thinking when updating Dell PowerEdge firmware.

With Dell hardware you can get the iDRAC to do firmware updates directly, either through the GUI, SSH, or the racadm client tool. Racadm can be obtained from support.dell.com. From there, here’s what I do, often in a PowerShell ForEach loop or pasting from an Excel spreadsheet.

First, I clear the old job log, which makes it easy to see my new jobs and their state, as well as guarantees any old jobs that haven’t run get deleted. Replace <iDRAC> with the IP or host name of the iDRAC itself, and <PASSWORD> with the password for the user.

racadm.exe -r <iDRAC> -u root -p '<PASSWORD>' jobqueue delete --all

Give that a minute to complete and the iDRAC should show no jobs:

From there, issue the command to pull the firmware from downloads.dell.com. This assumes that the iDRACs are properly configured with either a direct route out or a proxy. Having a direct path out — properly protected with firewalls or ACLs, of course — is easiest. It also enables you to set your NTP servers correctly so your SSL works right and your logs have the correct timestamps. #justsayin

However, if these aren’t options for you you might explore the Dell EMC Repository Manager (Google it, the link changes all the time), which allows you to download the updates and place them on your own web server or file share. I’d recommend a web server at this point, because only the most recent iDRACs can do modern SMB protocols, and you really need to have SMB1 off on your stuff.

racadm.exe -r <iDRAC> -u root -p '<PASSWORD>' update -t http -e downloads.dell.com -a FALSE

This tells it to download via HTTP, from downloads.dell.com, and the “-a FALSE” tells it not to restart the machine but just queue the updates for the next reboot. In contrast, if you have a machine that you’re okay with restarting right away you can flip that to TRUE. If there are updates you should see some jobs show up in the queue:

From here, just reboot the hosts. I like doing this as part of a VMware vSphere patch as well so that Update Manager does all the hard work of rebooting my clusters for me. Once that’s done the jobs should change to:

A couple other things to note:

I’m serious about getting NTP set up on your iDRACs, it’s important. People block it “because security” but security is exactly why it needs to be on. While you’re in there, set up syslog and turn SSL to TLS 1.2.
I usually run the Dell PowerEdge firmware update job once from the iDRAC GUI to see if there are any relevant updates. Then I know what I’m up against. You don’t have to schedule updates in the GUI, but it’ll show them to you. The update function moves around between versions a bit but is usually called “update/rollback” under the iDRAC settings.
If you do use a separate repository there are other flags you can reference to help you. Best way to get help is to ask the iDRAC for help, “racadm help update” (use the command format above).
If you have iDRAC SSH on you can log in that way and run these commands, too. I don’t recommend enabling SSH unless you really need it, though, it’s just attack surface that bad actors can use and compliance people get jumpy about.
You should watch the first host you patch to make sure everything goes as planned. For example, the early versions of 14G PowerEdge servers had some absolutely essential NIC updates that needed the host to be power-cycled, and that could have been disastrous if we weren’t doing staged rollouts. We ended up cancelling those and then orchestrating them with PowerCLI and racadm outside of Update Manager.
Don’t preload Dell PowerEdge firmware a long time before you plan to reboot. If the host crashes or you have a power outage it’ll update itself on reboot, and that’s just another complication you’ll have to face in the middle of an outage.
That said, don’t ever let a crisis go to waste.
If there are iDRAC updates (the iDRAC itself or the OS driver packs) those will apply by themselves automatically regardless of the “-a” flag. This generally causes a brief iDRAC outage and some weirdness during the update, which isn’t a problem if you know it’ll happen. If you check for updates using the GUI you’ll see this and can plan ahead.

As always, good luck.

End Anonymous Conference Feedback

Bob Plankers February 15, 2019

People Stuff

There’s a lot of talk lately about the terrible, horrible, sexist, racist, misogynist, and generally unconstructive feedback that presenters get at conferences:

This is on top of feedback where an attendee gives a presenter one star because the food wasn’t what they wanted, or they sat next to someone smelly, the room was cold, an “intro” talk wasn’t technical enough, or an “advanced” talk was too technical. Those things are out of the control of the presenter, and by giving the presenter a bad rating the attendee is jeopardizing that presenter ever being able to speak at the conference again. Not that I’m bitter, though it was pretty eye-opening to how juvenile and toxic the Linux community is when you say something they might not want to hear.

This absolutely needs to get fixed if conferences want decent, thought-provoking speakers of all genders, races, and beliefs. I propose a three-pronged approach:

Feedback that is inappropriate should not be counted into the ratings for a speaker. Giving someone two stars and saying in the comments that it’s a shame she’s married, yeah… no. While a conference organizing committee isn’t going to have cycles to read everything, allowing speakers to reject feedback back to them may be a way to deal with this.

Low- & high-rated feedback should require an explanation. What, exactly, did you not like about the presentation that you gave it one star? Or, the converse, what did you really like that you gave it 5 of 5? No more blind extraordinary ratings.

Feedback should not be anonymous, ever. Anonymous feedback encourages people to do and say things that they’d never say face-to-face. While I understand that names are a tricky thing sometimes for certain populations of people, being able to correlate a racist comment to a conference registrant is a key part of making conferences better.

Will this fix anything? Can we do it? Maybe. I’d like to see us start talking about it, though, because as it stands this cannot continue.

Out-of-Office Messages are a Security Risk

Bob Plankers February 3, 2019

People Stuff, Security, System Administration

Every once in a while I get asked why I don’t have an out-of-office message for my email or voice mail. Truth is, I’ll often monitor my email even when I’m out, though I often practice good operations discipline by not responding. Just as intermittent problems with computer systems are hard to deal with, a staff member that’s supposed to be gone but isn’t acting like it is just as confusing. Humans can, and should, drain-stop and remove themselves from clusters for maintenance, too.

Sometimes I’m really out of the office, though, crawling around in the backcountry wilderness or on an island somewhere. I’ll do it if I have to, but even then I don’t like setting an automatic response. There’s no way to do it that doesn’t leak information to a would-be attacker.

I’m out of the office. I’m saying that, just like my email, I’m probably not watching my computers or accounts closely so now is a good time to drain my bank accounts, install malware on my PCs, and social engineer my coworkers.

I’m gone between dates X and Y. You’ve got that long to defeat my security without any active opposition from me. The length of time also speaks to whether I’ve traveled somewhere, and they can break into my house unnoticed. This could cause evidence of a break-in, like fingerprints, to degrade, as well as allow for secondary damage, like rain coming in a broken window. If I’m the only one traveling this may also endanger others still at home in my house.

I’m hiking the Inca Trail to Machu Picchu. I gave it away, and on top of it I’m bragging, too. To quote Peter Quill from Guardians of the Galaxy, “what an a-hole.”

Contact somebodyelse@company.com with urgent needs. They now know more about the team structure and can do some social engineering. “Before he left Bob told me you could give me get an account on system X.” Plus you’re unfairly burdening someone else, and forcing people to make value judgments about urgency, too.

You might be saying “Bob, you’re nuts. I don’t have would-be attackers, I have customers.” I’ll agree with you, but I’ll also say that it isn’t like an attacker is going to advertise themselves. It’s a fine line. Use your head.

Set the autoresponse to the smallest group possible. In many cases you can narrow it down to coworkers, and/or have a different message for people inside your organization than outside your organization.
Only reply to messages that address you specifically in the To: or Cc: field, and only reply to the original sender. This helps prevent loops and extra email.
Only reply once a day per person. Chances are they can remember that for a few hours.
Test it by sending yourself some email from a different account.
Don’t tell people anything more than they need to know. Does everybody really need to know where you’ve gone and how long? Probably not. You’re just gone.
Recommend standardized alternatives if you have them, like “Please contact our Help Desk” or “Email the team list.” The personal relationship you have with your customers may not extend to others on your team, so don’t make assumptions. Besides, with you gone the team may be understaffed and dealing with personal, one-off requests will strain both them and the customer.
Make sure the team knows that just because someone says you said something doesn’t mean you actually did. Always evaluate every request independently. “Sorry, Bob is out, who is your manager? I need to call and have them email me authorization for this so we have a record of it.” Don’t be afraid to put something on hold until you get written confirmation. If it’s as urgent as they say they’ll find a VP or a C-level to clue you in. If you get any static from those VIPs don’t get flustered, be polite and tell them that it’s standard security practice to corroborate requests. Would you give keys to your house to random people who knocked on your door? No, no you wouldn’t.
If you’re gone be gone. I understand the urge to help, especially if you are watching your email and it looks like something bad is happening. Don’t reply, work through a team member. “Hey Joe, I just saw that email about system A that’s broken. I’m not going to do anything here because I just found some WiFi and checked my email, but if I were there I might look at X or Y first, that’s what the problem has been in the past. Good luck.” Do you want to ruin your vacation by having to own and work a problem? I bet not. Let the people at home handle it.
Tell the team to make a list of anything they have trouble with because you were gone. Don’t take any of it personally, this is a great opportunity for cross-training and documentation if you don’t have it. Banks often have a policy that people in sensitive positions be gone for at least two consecutive weeks to expose fraud and embezzling, among other things. Given the Internet it’s hard to cut a sysadmin off like that, but at least you can use it as a way to find gaps in processes and documentation.

If you are a manager of people I implore you to enforce the “if you’re gone be gone” policy with your team. Studies increasingly show that people need downtime as well as balance in their lives. Setting an example of this behavior, as well as being serious about the “we’ve got this” attitude while folks are gone, is important. Especially with the “Brents” of the world, if you’ve read The Phoenix Project.

And yeah, maybe it will stink with them away, but then you know what the team needs to work on.

Good luck!

Free, Like a Puppy

Bob Plankers January 24, 2019

People Stuff, Security, System Administration

I’ve found that things that are free of charge are often not a good deal.

TANSTAAFL, or “There ain’t no such thing as a free lunch.” You’re always paying in some way. Maybe the piece of hardware is marked up more to cover the development cost of the “free” software that comes with it. Perhaps it’s the drug dealer model, where the first one is free to get you hooked. Sometimes you’re the product, and the “free” thing is spying on you with the hopes of making more money from ads or sales later. Certainly nearly every “free” web service is structured that way.

Beyond monetary cost, though, you paying for things with your time. “Free” things often fall into the “good enough” category, and “good enough” can be a trap. Best-of-breed products might cost money but you make that back in time saved, services delivered reliably, and higher efficiencies. Things that are merely “good enough” trade on your time, relying on humans to make up the difference. Sometimes they trade on your customers’ time, too, with downtime and complications in delivering services. In not doing the whole job they force you to buy something more, whether it’s more software to fill a gap or more hardware to make up for inefficient operations. Or, if you can escape buying something you often make up the difference with processes. These all cost time, money, and add unwelcome complexity.

Even worse, things that are “free” can have serious negative value. At best, a tool is another application you need to install, support, back up, and keep updated, just like your HR system or a web application. In fact, many “free” tools are built on the same components and need the same amount of effort. It might even need commercial software licenses for the operating system or SQL server. All of this is overhead and doesn’t move you or your organization forward.

The need for licenses is often skirted by delivering the software as a virtual appliance. Ask yourself what incentive the purveyor of a free tool has to patch it, to make sure it can be backed up and restored, and to configure it securely? All those things cut into their margins. I’ve scanned quite a few virtual appliances with security scanners, and the results are scary. If these “free” tools aren’t stealing your data outright they’re making it very easy for bad actors to do it to you.

People in the open source movement, when discussing free software, often have to delineate what kind of freedom they are talking about. “Free as in beer” meaning no charge, but the source code isn’t necessarily open. “Free as in speech” meaning that the source code is open. When it comes to the “free as in beer” tools we’ve been thinking about here I often use the term “free as in puppy.” That cute puppy dog you adopted from a friend is going to cost you thousands of dollars and hundreds of hours of your life, time you will never get back again. Are you sure you want to do that to yourself and your organization?

Your mission, from now until the end of your life, is to treat everything that’s “free” as if it’s a Trojan horse. If you can’t say no to something outright ask some questions:

Can I call support for this product?
Does it collect data? What does it collect? Where does it send that data? How does it secure the data in transit? Is the data anonymized? Who can see that data? Can I disable or block that without impacting the functionality?
How do I patch it? Where is the documentation for patching? There are vulnerabilities discovered in every OS every week. When was the last patch release?
What operating system does it run on? Do we have expertise in that operating system? Is that operating system still supported by the vendor?
Does it need licenses of any sort to run? OS? SQL? Java? Remember you can’t run certain Java versions commercially anymore without a license. Are there license levels or limits to the “free-ness” of the product?
How do I get to it? Web interface? Does it need Flash or Java? Can I install my own SSL certificate? Can it be restricted to TLS 1.2 or newer? What browsers does it support?
Is the host-based firewall enabled? Is it configurable?
Is the SSH server key generated at first boot so it is unique? Can we disable SSH?
What other services run on this appliance?
Can it send its logs to a SIEM? How? Does it support more than one target? Does this include the application logs?
How does it do authentication? Are there password complexity guidelines?
How do I back up the configuration? How do I restore it?
Can I replicate it to a DR site? Are there latency requirements? Have you tested it across a WAN?
What security guidelines does the product conform to? Is it PCI compliant? NIST 800-53? Does it implement the Center for Internet Security’s benchmarks?
Do I have to use it as a virtual appliance, or can I install it on an OS that we installed and secured?
Are there any other ways to achieve what this tool does? Are there other products or methods on the market? Is there a way to do this with the CLI? Are there Powershell modules? What SDKs are available, and for what languages?
Will there be ROI for the time and effort invested here? This is a question for yourself and your own team.

“That’s a lot of questions,” you might think. But hey, there might be a lot of bad dudes in the belly of that wooden horse, too. Better find out before you bring them inside the walls.

Good luck.

Retrieve an SSL Certificate from a Server With OpenSSL

Bob Plankers November 26, 2018

System Administration, Virtualization

I was setting up VMware vRealize Automation’s Active Directory connections the other day and I needed the public SSL certificate for the AD DCs to authenticate correctly. You can use OpenSSL to get that information.

I used a Linux shell but this should be do-able from a Mac or with OpenSSL installed on Windows, too.

If you wanted to read the SSL certificates off this blog you could issue the following command, all on one line:

openssl s_client -showcerts -servername lonesysadmin.net -connect lonesysadmin.net:443 < /dev/null

In this case you’ll get a whole bunch of stuff back:

CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = lonesysadmin.net
verify return:1
Certificate chain
 0 s:/CN=lonesysadmin.net
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFWDCCBECgAwIBAgISA/05rMV6H+0LKP7uo3EE2F5zMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODExMTIwNTEzMTBaFw0x
OTAyMTAwNTEzMTBaMBsxGTAXBgNVBAMTEGxvbmVzeXNhZG1pbi5uZXQwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC22CK5EPHggy7q6qgqiYObGumi4b6X
DV/xKXauS5P2w2zjUFnHO01KRPDQ2owrc4opNRbngqanBI6llWTlBTrMCJSCa8sQ
uE6PEp7Hs94y1xHKKWQ+Lkk8ha99E50plAAU0CY3M7qqJ5Js5Q6L+MZ94H7/4Vvr
c0Ojiu6iXug5YmeNteqE3gzJvQcNoEk8js4HUUuH1FJNm1dLtKFY1/NCut6M39zC
1QOh7YfuGj/DgZD0le24SlBFYgv/2kDwVdPesIktyuV0aPZ+gihLFAjQotCvfhZ1
lQ8/+n9gFlG0vpjVdv8ZurixeyUykVk/Xq9HZLPwaDCubJ2NT+tZoqyXAgMBAAGj
ggJlMIICYTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFFMnifCC04NozOYpDuRzN16P
+WuOMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEB
BGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZy8wGwYDVR0RBBQwEoIQbG9uZXN5c2FkbWluLm5ldDBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3AHR+2oMx
rTMQkSGcziVPQnDCv/1eQiAIxjc1eeYQe8xWAAABZwaNaRwAAAQDAEgwRgIhAOWY
Xz2ucXa1Src4Ec6cBR6C6TNttfkGGR9mzTGM/SvZAiEA+EgD4cX9Wl+fqctk3iB8
ho/efgo3Urh9BPPQYefb4zQAdQApPFGWVMg5ZbqqUPxYB9S3b79Yeily3KTDDPTl
RUf0eAAAAWcGjWrZAAAEAwBGMEQCIGmIhmJ1x68Z0s32ZTTv6NBD1pzRaSTo92t8
mybQWsnsAiAg+R3O4ruTA/ao1OZfjlg5GG8/+9Z6qtTGLF0bK+cyDTANBgkqhkiG
9w0BAQsFAAOCAQEASsi4FWVLjkLG04BgtnPWyxWoPTFY16MDTLpat6Clf9oQTjb4
KcpNn9tYlLyv9NWNYnQiD351IvDAoPG1Lk4iGdQlo1kFNlrjyozZlQGax7g1XuXX
9OasSVK5aJtIGStR7J2NQsgbLgbY9AajC9BWE7lCPE9AsRIDj96nN3DgbL8hng3d
2G2CBUEYC9+FqnhpqNUXuogLNr9SUm6AODLsRMGoQ0lXwLmjrb++tKuEn55SlHHF
GvDHKB/qCfjLFpByk05J49v2qDBJcxmOkyTHyi9TnDBplWDRCHO99J6qgMorWma3
UavqxAHS6Q3h/kjsjDhBo90o2fDr+gKKRBiTmA==
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
Server certificate
subject=/CN=lonesysadmin.net
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
SSL handshake has read 3260 bytes and written 398 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 3B67BD8D78293E6C2CD87E192316DF2B0DD5B8D8D3E0209DD2A2F2CBE0D8298C
    Session-ID-ctx: 
    Master-Key: BA7C4F7737DA489457285514FA66E935EAD13D4D8DAADA7577917A9B4564120759535FCF76C6616CC96108C375DA015A
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 9b 60 9e 06 36 26 95 27-0a b5 3e ba e2 9f e2 5c   ...6&.'..>....\    0010 - 71 f1 c4 12 2f 73 60 5e-ed 3b 19 fd af 48 51 4d   q.../s^.;…HQM
    0020 - 85 47 93 5b b4 83 45 ef-04 15 ba 59 85 96 eb c1   .G.[..E….Y….
    0030 - 70 da e2 6f c4 f5 99 b5-ed c0 c2 6b 67 73 85 4e   p..o…….kgs.N
    0040 - 3e f1 6f e2 3c 5c f9 1f-e9 d3 8b c1 96 53 ea b2   >.o.<.……S..
    0050 - dd a8 e9 0e 20 5c a5 de-c9 80 cc c6 35 62 c1 51   …. .…..5b.Q
    0060 - c0 64 b3 2f ca eb 15 97-2a cd ef 51 8e 5f 21 32   .d./….*..Q.!2    0070 - 4b d9 f9 2e ba ec b1 e5-06 cb dc 57 ab 1d 23 28   K……….W..#(    0080 - 76 41 9c 79 e4 05 23 68-c4 2c 0c f1 46 df 55 01   vA.y..#h.,..F.U.    0090 - 0e 68 d8 83 53 e1 8d 02-18 d4 b0 3d fc a6 03 9a   .h..S……=….    00a0 - 2c 68 88 79 91 4b c9 ba-47 40 b4 aa d3 fb 17 e5   ,h.y.K..G@……    00b0 - d5 36 f2 45 10 70 dd c4-1e be 69 6a d0 88 e1 a7   .6.E.p….ij….    00c0 - ac 5f df ef b1 e7 bc be-42 06 8f 8c f3 82 95 5c   .……B……\
Start Time: 1543255454Timeout   : 300 (sec)
    Verify return code: 0 (ok)
DONE

Just prune out everything that isn’t between a “BEGIN CERTIFICATE” and “END CERTIFICATE” line:

And ta-dum! you can paste that into whatever needs it. Some stuff might need it in reversed order, so if it doesn’t work this way just rearrange it.

Don’t forget to use the correct hostnames and ports! If your AD DC is called dc-01.goatrodeo.org and the global catalog is on port 3269 it’d be:

openssl s_client -showcerts -servername dc-01.goatrodeo.org -connect dc-01.goatrodeo.org:3269 < /dev/null

Good luck!

What You Need to Know About Upgrading to an iPhone Xs or Xr

Bob Plankers November 21, 2018

Hardware

I just got a new iPhone Xs Max. I had an iPhone 6s which I liked a lot, but it’s been a few years and with more travel I thought I’d enjoy having a better device with me. There are a few things that bit me in the duff.

Some two-factor authentication (2FA) apps like Duo or Google Authenticator store their data in the iPhone Secure Enclave, which isn’t backed up to iCloud or via iTunes. That means that when you switch devices (or if you lose your device) you could lose access to your accounts, or it’ll be a serious pain to regain access (which is the point of 2FA). So don’t trade in your old phone until you’ve re-registered everything!

@funnelfiasco suggested to me that I switch to Authy, which allows backups and multi-device access. @millardjk suggested keeping screenshots of the registration QR codes, which could be a security issue but would help immensely for reregistering between apps. @vcixnv suggested that SAASPASS was a solution for him, instead of Authy. Thanks Ben, Jim, and Britton!

(I went with Authy by the way, and I used the Google Authenticator features rather than Authy-specific stuff in what is probably a futile attempt at avoiding lock-in).
No button is kind of a pain in the duff. Get the home screen by swiping up from the bottom bar (the “home bar”).
Switch apps by swiping left and right on the home bar, or by swiping up from the bar to the middle of the right side of the screen. Yes, that’s convoluted. I miss the button.
Summon the control center by swiping down and towards the center from the top right (same as with iPads now). You can customize those controls in Settings -> Control Center -> Customize Controls. That might be new in iOS 12, or not, but I hadn’t noticed it.
There was a feature called “Reachability” on the old phones, which will bring the top of the screen down. On my new phone it was off. You can enable it by going into Settings -> General -> Accessibility and flipping it on. Once it’s on you can swipe from just above the home bar down to move the screen down. It’s still a bit kludgy, but hey.
So I tried to shut the phone off, and discovered they moved it from the side button. Much cursing ensued. You now have to hold the side button and one of the volume buttons together for two seconds. It’s also the SOS function so when you do that you’ll have to enter the passcode again. I’m fine with that, I don’t want someone to point it at my face and gain access to my phone.
If you want to take a screenshot it’s the side button and the volume up. If you tap on the screenshot afterwards you can mark it up or delete it, which is nice.
Make sure that cellular data is on. Mine got shut off in the conversion. If your provider allows it make sure you’re using LTE for both voice and data. This is important on CDMA networks because the older modes didn’t allow for simultaneous voice and data. That’s a bummer if you’re using your phone as a hotspot. Settings -> Cellular -> Cellular Data Options. While you’re in there you might check out the WiFi calling options, too.
Face ID is interesting. I think I like it so far, but it took a little to get used to it. My wife and I set our phones up so we can each unlock them without knowing each other’s passcodes (this is 95% being able to see a recipe on the other person’s phone while we’re cooking, and 5% me updating her phone). Instead of registering a fingerprint, Face ID allows for an “alternate appearance” which we just use for the other person. Settings -> Face ID & Passcode -> Add Alternate Appearance.
With Face ID the auto-lock gets set to 30 seconds. Yuck. While I was in Settings -> Display & Brightness I also disabled Raise to Wake. If you just tap the screen it’ll wake up. I also set Settings -> Notifications -> Show Previews to “When Unlocked.” I don’t like others being able to see my incoming communications.

Overall, do I like it? Sure. The Xs Max is large and expensive and it’s taking me some time to adjust to the size (I’m a week into it), but I think I’ll like it overall going forward. I just wish some of this was on the “Welcome to iPhone” setup spiel, and not just Siri and Apple Pay.

CODE Keyboard

Bob Plankers November 6, 2018

Home Office, People Stuff, System Administration

“You spent $150 on a keyboard?” – My wife

There are two kinds of people in technology: those with an opinion about their keyboard, and everybody else. I happen to be one of the first.

Buckling Spring image courtesy of Wikipedia.

I grew up using the IBM Model F and M keyboards. They have a spring in the key switches that buckles as you press down. That gives you two things: a prominent clicking sound from the keypress, and solid tactile feedback from the key. You definitely know when that key switch actuated.

Years ago I had to give up my Model M keyboards. They’re built to last but it was getting harder to find working ones, it was getting inconvenient to adapt them to USB from PS/2, and a case of carpal tunnel made it painful to use a keyboard that required a decent amount of force to type. This also pleased my coworkers, who didn’t particularly like the stream of loud clicking when I was in the office. And so I settled on a series of Dell keyboards, mostly because we had some sitting around. The multimedia controls on the newer Dell Business keyboards are nice, and I’ve been using those for a while now.

“Does it do cool things?” – My six year old daughter

In a few weeks I’m not going to have coworkers within 50 feet of me, and my old keyboards are getting a little, well, old. So I thought I’d treat myself to a new keyboard. Over the last couple years I’ve been lurking in the community around keyboards, marveling at the incredible love that people pour into the devices at their fingertips. In particular, Massdrop has a quite the stream of interesting keyboards and customizations, many available for purchase. There are cheaper options there but I don’t like ground-effect lighting for my keyboard enough to spend $500, though.

Turns out you can buy a faithful clone of the IBM Model M from Unicomp, but I think I’m past the mega-clicky stage of my life. I don’t want people to hear all that when I’m on the phone. So after looking around I decided on a 104-key CODE Keyboard, which is a collaboration between Jeff Atwood of Stack Overflow fame and WASD Keyboards. You can choose the switches that are in it so you get exactly what you want for noise, feel, and actuation pressure. The keys have backlighting, which is great. The keyboard weighs a couple pounds, so you can defend your home office with it if you need to, and it has big patches of rubber underneath so it does not move. It’s got a standard USB cable (micro to A), so you can replace it or customize it, and a bunch of routing options underneath. And best of all, it’s simple & clean.

It’s got six DIP switches on the back to customize it if you are a Mac, Windows, or UNIX person (if you’re used to a Sun keyboard that swapped Ctrl and Caps Lock). I flipped the sixth switch so that the keyboard Function key can do the multimedia controls (versus an OS “menu” key). If you want to customize it further you can just order a WASD v2 keyboard and customize it fully, from a variety of languages and layouts to what color each key is. I liked the compromises and the LED backlighting in the CODE model, but I can order new keycaps in the future if I want.

“I AM A BAT. I FLY.” – My three year old son, unfazed by a new keyboard

Best of all, I was looking for a reason to try it out, so I wrote this. It’s definitely a different feel than my old keyboard, but that’s what I wanted. I like it so far. At the beginning here I was doing a lot of double capitalization (WRiting THings Like THis), but 600 words in that seems to have cleared up. I think this keyboard and I might get along just fine.

Now I need to find an amazing mouse to go with it. Thoughts?

Joining VMware

Bob Plankers November 2, 2018

People Stuff, Professional Development, Virtualization, VMware

“We changed again, and yet again, and it was now too late and too far to go back, and I went on. And the mists had all solemnly risen now, and the world lay spread before me.” – Pip, Great Expectations

Growing up the son of a firefighter and homemaker, I was fortunate to have been given the opportunity to go to college so many years ago. So in the autumn of the release of Windows 95 I left my childhood home to go to school at the University of Wisconsin – Madison. At four hours by car the UW was far enough away from my parents that they wouldn’t stop in randomly, but it was close enough that I could go home easily. I never really went home, though. Sure, I’d go visit, but my home became Madison, and I dug in. And while my parents helped with my tuition, room & board was solely my responsibility. I got a job, hired at the UW-Madison Help Desk to do phone support for the dial-in modem pool.

Information Technology wasn’t a career path when I was in high school, at least according to the school guidance counselor who told me I was going to be a chemical engineer, and that was that. All engineering students go through the first sets of classes together, though, and along the way I heard about Electrical & Computer Engineering. Took me about 12 seconds to switch. The grass is always greener, it seems, and it didn’t take long for me to figure out that I liked the software side more than hardware. The overlap with Computer Science seemed a natural path.

Fast-forward a few years. I’d been promoted out of the Help Desk. I was running giant AIX systems for our PeopleSoft implementations, and I was wondering what was next in my life. The work I was doing was so much more interesting than school, and it was the path I wanted to be on. I liked the UW, I had lots of friends there, and the people I was working with and for had interesting problems to solve. Above all, it was safe and familiar. My father died in 2001 and that left me adrift and with a case of PTSD, so when the UW offered me a real job, with real pay and real benefits, I signed on.

23 years later I’ve been fortunate to have worked with some of the brightest (and interestingly enough, fastest and strongest) folks around. I’ve been able to reinvent my job a few times, as new technology comes along to reshape the landscape. Landscaping in higher education involves a lot of hard work, overcoming inertia of silos, culture, and incredible fear of change. It requires immense amounts of patience. It has worn on me, as I’d seen my father’s job as a first responder wear on him, turning us into sarcastic, bitter, angry people. I grew more and more like the mythical Sisyphus, destined to roll rocks up hills as punishment for offending self-appointed gods in non-specific ways.

I’ve been thinking about moving on for a while now. I don’t want to turn into my father, and I cannot keep rolling the rock uphill for 20 more years. I’ve talked to a number of friends that have made the leap to vendors, all of which told me, nicely, to shut up and do it. I clearly enjoy technology, but I also enjoy speaking and writing about it to help others understand more. I’ve been active in the VMware community for years. With all of that I’ve been envious of the work the VMware Technical Marketing folks do in all these spaces, getting paid to do the things I basically do as a hobby.

With two small children I’ve been hesitant to take a position with a lot of travel, though, and I’m very fortunate to be in a spot where I could take some time to make sure where I was going is a very good fit. That said, it took almost no time for me to respond when I was asked to consider applying for a position at VMware, in the Cloud & Platform Business Unit’s Technical Marketing group. I am the secret Mike Foley’s been dying to reveal on Twitter, and I’m very excited to work with him, Adam Eckerle, Niels Hagoort (who just joined as well) and all the others that produce such great content and understanding for VMware customers.

I start at VMware in early December and for the first time in a long time I feel again like Pip in that quote above, excited and nervous at the possibilities that lay before me.

Fixing X11 Forwarding Over SSH and with Sudo

Bob Plankers July 6, 2018

System Administration

X11 forwarding over SSH not working? Not setting $DISPLAY correctly in your shell? Having problems with X11 and sudo? Yeah, me too. Total pain in the duff. Here’s what I do to fix it. I’m thinking about Linux when I write stuff like this but a lot of this has worked on AIX and Solaris, too.

Make sure your SSH client supports X11 Forwarding and that it’s turned on. I use SecureCRT but I know it works in PuTTY as well. Once you turn it on in your client & save the settings you will need to reconnect, the forwarding is established with the connection.
Ensure xauth and xterm are installed. You need xauth for this to work, and xterm is a lightweight way to troubleshoot this stuff (just run “xterm” at a shell prompt and a window should pop open).
If you are using a command-line client, or forwarding across multiple hosts, is X11 forwarding enabled in your ~/.ssh/config file? Add “ForwardAgent yes” and “ForwardX11 yes” to it. You can also force it with “ssh -X user@host” when you connect.
Do you have an X Windows server running on your desktop PC? I use Windows on my desktop and I use VcXsrv. Make sure it’s started and running. VcXsrv asks me how I want to run it, I always choose “Multiple windows,” set the display number to -1 to let it choose, and start no client. You can futz with the rest once you know it’s working.
Is your $DISPLAY variable being set but you get errors? If so, that’s usually not forwarding, that’s something on your PC. Check your $DISPLAY with “echo $DISPLAY” at a prompt. It should have something in it like “localhost:10.0” or “localhost:13.0” or so. Does your X Windows server software (VcXsrv) have permissions? If so, set them wide open (allow all hosts to connect).
On your SSH server do you have “X11Forwarding yes” and “AllowAgentForwarding yes” in sshd_config? If it’s commented out uncomment it and restart the SSH daemon (“service sshd restart” works on a lot of distros).
Is your home directory writable? When you log in it’ll need to create an ~/.Xauthority file and if it cannot do that you’ll have problems.
Is your ~/.ssh directory writable and correct permissions? It should be owned by your user and chmod 700. Things in it should be chmod 600.
Is there an old ~/.Xauthority file sitting there? Try removing it and logging in again.
Did you disable IPv6? If you run “sysctl net.ipv6.conf.all.disable_ipv6” and it comes back as 1, or “lsmod | grep ipv6” shows nothing you might have IPv6 disabled. Turns out OpenSSH hates that and has a very passive-aggressive way of showing it. Add “AddressFamily inet” to your sshd_config and restart the daemon. That forces it back to IPv4 only.
Are you trying to run something as root using sudo or su? Getting “X11 connection rejected because of wrong authentication?” That gets funky because of permissions with xauth. There are lots of tricky fixes with xauth but I’ve just found copying my .Xauthority file to my target user works great. Then you can “sudo xterm” with impunity. You might try avoiding “sudo su -” as the hyphen wipes your environment out, and along with it your $DISPLAY. Just try “sudo -u targetusername command” instead.

sudo cp ~plankers/.Xauthority ~root/.Xauthority

If you’ve gotten this far and you’re still not able to run ‘xterm’ and have it pop a window open I’m surprised. Try SSHing with debugging on, “ssh -v -X user@host” and see if it tells you what’s wrong. Add more “v” to increase the debugging level, like “ssh -vv -X user@host.”
What do the logs say when you connect to the server? A lot of times when there’s something wrong it’ll put something in the logs about what it is.
Absolute vanilla installs of Linux distributions usually work fine. As a last resort try a VM running a stock installation of something like Ubuntu and see what happens.

Good luck! I hope at least some of this helps.

Fixing Veeam Backup & Replication Proxy Install Errors

Bob Plankers July 2, 2018

Virtualization, VMware

Every once in a while I struggle a little to add a new Veeam Backup & Replication hot-add proxy. If you’re like me and seeing proxy install errors maybe some of these will fix you up. This is what worked for me on Windows Server 2016 when I was getting error 0x00000057, “Failed to create persistent connection to ADMIN$” and some other unhelpful messages.

If you’re using a hardened Windows installation all bets are off, since the goal of hardening is to intentionally disrupt remote access. I’d get it running with as close to a stock Windows installation as possible and then work from there if you need to secure things further. There are also ways to manually install the Veeam Transport Service that might be more helpful.

You might want to consider taking a snapshot before this work, so when you discover what fixes the problem you can revert the snapshot and just implement the fix cleanly.

First, try specifying the username as the full “DOMAIN\Username” format when you add it to the Backup & Replication console. Don’t use the “.\username” format and don’t omit the domain part itself. If you are using local accounts you’ll want to specify “SERVERNAME\username” instead, using what the proxy knows as its name. This alone fixes 90% of the issues I’ve seen.
If you aren’t using the Administrator account (and it’s a good idea not to) does the account you want to use have Administrator rights on the proxy VM, and the correct password? I sometimes forget to add the domain service account I created to the local administrators group.
Check to see if you can reach the administrative shares on the proxy VM. Do this from the Backup & Replication main backup server itself by browsing to \\COMPUTERNAME\\Admin$ using the credentials you’re going to use for Veeam. This may mean you need to use “net use” to map it so you can specify a different username. If that works you should see the Windows directory on the remote computer.
Didn’t work? Is the firewall enabled? For troubleshooting try adding an explicit “allow any” rule for all traffic to & from the backup server. If that makes browsing to Admin$ work then make sure you have rules to permit traffic between the proxy and the other proxies, and the proxy and the main backup server. Note that you can test this by just shutting the firewall off, but don’t do that unless you’re protected in some other way (hardware firewall, etc.).
If the firewall is disabled and you still cannot browse can the backup server ping the proxy? Is there another firewall between them that’s denying traffic?
If the firewall is disabled, they can ping each other, and you still cannot browse have you disabled remote UAC on the proxy VM? Open an administrator-level command prompt and run:
```
reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\system /v LocalAccountTokenFilterPolicy /t REG_DWORD /d 1 /f
```
Reboot the proxy VM and try again. At this point you probably can browse to Admin$, and you should take a moment to make sure your firewall is on and everything is secured again. If you still can’t get in I’d look at more fundamental issues, like time synchronization and DNS.

Good luck!

Easy Dell PowerEdge Firmware Updates, 2019 Edition

Like this:

End Anonymous Conference Feedback

Like this:

Out-of-Office Messages are a Security Risk

Like this:

Free, Like a Puppy

Like this:

Retrieve an SSL Certificate from a Server With OpenSSL

Like this:

What You Need to Know About Upgrading to an iPhone Xs or Xr

Like this:

CODE Keyboard

Like this:

Joining VMware

Like this:

Fixing X11 Forwarding Over SSH and with Sudo

Like this:

Fixing Veeam Backup & Replication Proxy Install Errors

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: